Generalized Biwords for Bitext Compression and Translation Spotting: Extended Abstract

The increasing availability of large collections of bilingual parallel corpora has fostered the development of naturallanguage processing applications that address bilingual tasks, such as corpus-based machine translation, the automatic extraction of bilingual lexicons, and translation spotting [Simard, 2003]. A bilingual parallel corpus, or bitext, is a textual collection that contains pairs of documents which are translations of one another. In the words of Melamed [2001, p. 1], “bitexts are one of the richest sources of linguistic knowledge because the translation of a text into another language can be viewed as a detailed annotation of what that text means”. Large bitexts are usually available in a compressed form in order to reduce storage requirements, to improve access times [Ziviani et al., 2000], and to increase the efficiency of transmissions. However, the independent compression of the two texts of a bitext is clearly far from efficient because the information contained in both texts is redundant. Previous work [Nevill-Manning and Bell, 1992; Conley and Klein, 2008; Martı́nez-Prieto et al., 2009; Adiego et al., 2009; 2010] has shown that bitexts can be more efficiently compressed if the fact that the two texts are mutual translations is exploited. Martı́nez-Prieto et al. [2009], and Adiego and his colleagues [2009; 2010] propose the use of biwords —pairs of words, each one from a different text, with a high probability of co-occurrence— as input units for the compression of bitexts. This means that a biword-based intermediate representation of the bitext is obtained by exploiting alignments, and unaligned words are encoded as pairs in which one component is the empty string. Significant spatial savings are achieved with this technique [Martı́nez-Prieto et al., 2009], although the compression of biword sequences requires larger dictionaries than the traditional text compression methods. The biword-based compression approach works as a simple processing pipeline consisting of two stages (see Figure 1). After a text alignment has been obtained without pre-existing linguistic resources, the first stage transforms the bitext into a biword sequence. The second stage then compresses this sequence. Decompression works in reverse order: the biword sequence representing the bitext is first generated